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LANGUAGE 
EE TINGS 
D CONFERENCES 



March 3-6. Convention of Teachers of English 
to Speakers of Other Languages, 10th. 
New York, NY. 
March 14-15. North American Conference on 
Afro-Asiatic Linguistics, 4th, and 
American Oriental Society. Philadelphia, 
PA. 

March 21-24. Association for Asian Studies. 
. Washington, D.C. 
^ March 25-27. Conference on Composition and 
y Communication. Philadelphia, PA. 
/ April 1-4. Northeast Conference on the 

Teaching of Foreign Languages. New York, 
NY. 

April 3-5. Annual Meeting of the Linguistics 
Association of Great Britain. Edinburgh, 
Scotland. (Write: J. Christie, Dept 
Linguistics, 15, Buccleuch PL, Edinburgh 
EH8 9LN, Scotland). 
April 5-9. International Conference of 
Nordic and General Linguistics, 3rd. 
Austin, TX. 
April 6-9. Acoustical Society of America. 

^ Washington, D.C. 
April 9-10. College English Association. 

Cincinnati, OH. 
April 15. [Deadline for abstracts for LSA 

Summer Meeting]. 
April 15-17. Conference on African Linguis- 
tics, 7th. Gainesville, FL. (Write: 
P.A. Kotey, Ctr African Studies, 470 LGH, 
Un1v Florida, Gainesville, FL 32611). 
April 19-23. American Educational Research 

Association. Las Vegas, NV. 
April 20-24. Congress of the International 
Association for the Study of Italian 
Language and Literature, 9th. Palermo, 
Italy. (Write: Robert J. Clements, 
AISLLI, Rm 701, Main Bldg, New York Univ, 
Washington Sq, New York, NY 10003). 
April 22-24. University of Kentucky Foreign 

Language Conference. Lexington, KY„ 
April 22-24. Southwest Areal Language and 
Linguistics Workshop, 5th. San Antonio, 
TX. (Write: B.L. Hoffer, Dept English, 
Trinity Univ, San Antonio, TX 78284). 
April 22-25. Chicago Linguistic Society. 
Chicago, IL. (Write: Chicago Linguistic 
Society, Goodspeed 205, 1050 E 59th St, 
Chicago, IL 60637). 
May 6-8. Conference on Perspectives on 

Language. Louisville, KY. 
June 21-26. Conference on the Psychology of 

Language. Stirling, Scotland. 
June 2B-July 2. International Conference 
on Computational Linguistics. Ottawa, 
Ont, Canada. 
July 26-31. Philippine-American Communi cation 

Conference, 1st. Manila, The Philippines. 
August 17-19. World Congress of the Inter- 
national Reading Association, 6th, 
Singapore. 

August 26-31. World Congress of Phoneticians, 

3rd, Tokyo, Japan. 
August 28-30. European Linguistics Society. 
Salzburg, Austria. (Write: G. Drachman, 
Institut FOr Sprachwissenschaft der 
Universitat, Imbergstrasse 2/III, A-5020 
Salzburg, Austria). 
September 1-4. International Phonology 
'(UNCLASSIFIED) Meeting, 3rd. Vienna, Austria. 
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TIME'S c^UNNING 



OUT! 



You'd better 
hurry if you 
want to enter 
the CMI ESSAY 
CONTEST or the 
CLA ESSAY CON- 
TEST this year! 

The deadline 
for submitting 
entries to the 
CMI ESSAY CON- 
TEST is 26 March 
and the deadline 
for the CLA ES- 
SAY CONTEST is 
19 March. 




For complete information, see The NSA 
Technical Journal, Vol. XX, No. 4, Fall 
1975. Or call the following: 

CMI Contest:! J or P.L. 86-36 

CLA Contest: 



(UNCLASSIFIED) 




Comments, anyone? 



Mr. Phillips was Chief of the ADVA- 
GENS Joint Mechanization Group* which 
conducted the first tests of an AG-22- 
l.i-Ua d&tiez in I960 and 196l\~ 



also members of the group, 'me specif 
ic AFSAV-D/311 tests Mr. Phi Uins 
mentions were carried out by \ 



[were 



Puzzle. &an&, mathematician*, cJiyptanalyAtAl 
H&ie.'6 an oppontxmJUby you won't want to mi&Al 
Tke Ctyptomathematic6 In&tUute (CMI) kcu> a 
number oi copies oi PEUNEV PUZ2LES, a collec- 
tion oi 6S puzzle* by CMI '6 executive dJLh.ec- 
tor, Molten Penney. While the supply IomU, 
a ihte copy oi this ia&cinating uncla^iUed, 
book will be tent to everyone joining INI. ' ' 
To join, tend yovJi name, social 4 ecuhyOty^tum- 
ben., organization * telephone num ber, and $3.00 
pn. 7976 due* to: \ | CMI Trea6uA2A, 

SO 3. 



. (c) 

6-36 
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An Objective Approach to 

SCORING TR ANSLATIONS 

I IP16 



Reprinted from QRL (Quarterly Review 
for Linguists), November 1973 

Author's note : The philosophy underlying 
the translation grading system described in 
this paper has been developed and applied by 
Emery Tetrault and myself* with many valuable 
suggestions from our colleagues on Profes- 
sional Qualification Examination (PQE) Com- 
mittees and from other Agency linguists . My 
use of the pronoun "we" re fleets this collabo- 
ration. I personally take full responsibility , 
for presenting our findings here. 

* 

Translation as an intellectual activity has 
been practiced since antiquity for practical as 
well as esthetic reasons, but even today we 
have reached no consensus on whether it is even 
possible to fully "bring over" from one lan- 
guage to another the contents of a fairly pe- 
destrian piece of prose, let alone fiction or 
poetry. We debate the merits of machine trans- 
lation as a precursor to a truly "scientific" 
discipline or argue whether a "free" transla- 
tion of a Greek lyric poem does not better 
catch the flavor of the original than one which 
adheres rather closely to the syntax of the 
original and results in an awkward (i.e. "lit- 
eral") rendering in the target language. In 
short, the domain of translation has not been 
really determined. Our own work never demands 
(fortunately) the highly creative product of 
the artist translating Homer, Dante, or Pushkin 
but it does require something more than a purely 
mechanical approach; good translators have a flair 
for couching in idiomatic English the thought con- 
tent of the source-language passage. Poor trans- 
lators, on the other hand, seem never to master 
this complex operation regardless of how well 
they may know the source language or how well 
they may write English when not translating. 

If we had only the two kinds of translators 
to judge we could sort them into sheep and 
goats and have done with it. The fact is that 
most aspiring translators are neither brilliant 
nor hopeless, but fall somewhere in between, 
and when professional certification is at stake 
we need, in the interest of justice to the can- 
didate as well as to the system, reasonably 
objective criteria for making a final decision 
on or icUZ. 

Lacking such criteria in our early attempts 
to evaluate translations, we ended too often 
with intuitive decisions not transferable to 
similar cases involving the same source lan- 
guage; nor was it even possible to compare in- 



tuitive judgments across language boundaries, 
in source language-to-English situations. 

Over the past 2 or 3 years my colleagues and 
I have developed a way to score translations 
which may obviate this problem to a large ex- 
tent even though our results have admittedly 
been far from perfect (total "objectivity" in 
grading any kind of connected text is of course 
impossible) . Our first large-scale use of the 
system, which I will describe later, was with 
the Russian PQE. We have subsequently tried it 
in a number of other PQEs involving several 
languages, mainly Indo-European, but also from 
other families. The results have been encour- 
aging enough in both instances for us to recom- 
mend its use in the PQE Handbook. 

Subjective Approaches to Evaluating Translations 

One approach to scoring translations is in a 
sense historical in that the evaluator tries to 
guess the reason for the translator's mistake and 
in so doing draws unwarranted inferences. This 
attempt may be useful when the time comes to 
counsel the failing or borderline aspirant and 
the problems can be talked out. Unfortunately, 
though, the history of an error is irrelevant to 
the rendering of a particular text and whether 
the aspirant should have learned a particular 
grammar rule in a course is beside the point. 

A second method in wide use is to decide ar- 
bitrarily that a certain kind of mistake (for 
example, failure to recognize proper nouns and 
inappropriate translation of them) is sufficient 
cause to fail a paper. This syndrome is evident 
among longtime subject-matter experts or special- 
ists in some area of the world who see examina- 
tions solely in terms of their immediate inter- 
ests. 

One example of each approach should suffice. 
In the first case, a would-be translator missed 
a trickly negative conditional construction in 
the first sentence of a fairly long paragraph 
which contained other negative and conditional 
elements much less difficult. In the interest of 
coherent discourse the translator overrode the 
signals which "he should have learned" and pro- 
duced a sensible but wrong text. A second trans- 
lator decided that the name of a group of is- 
lands, which historically means "fishing plat- 
form," should be so translated. Not only did 
he fail to recognize the proper noun, but he also 
saw the context of an entire paragraph in terms 
of the translation, and produced a peculiar 
text in English. 

The evaluators of the first passage initial- 
ly decided that the paper should fail because 



March 76 * CRYPTOLOG * Page 5 



UNCLASSIFIED 



DOCID: 4009730 



UNCLASSIFIED 



the aspirant did not know enough of the grammar 
of his language, regardless of his performance 
in the rest of the examination. (Fortunately, 
we reconsidered the translation as a whole and 
gave the paper a marginal pass.) The second 
faux pas made in an early PQE apparently caused 
such anguish to the evaluators that they failed 
the paper mainly on this point. 

I hasten to add that none of the above is 
meant in a caviling spirit. Translations, wher- 
ever and by whomever done, have ordinarily been 
assessed by these or other equally subjective 
methods. What we propose here is an attempt to 
bring a measure of uniformity to grading trans- 
lations so that intuition and idiosyncrasy play 
a minor role and some kind of objective standard 
is established. The framework and detail of the 
proposed system is described below. 

Linguistics and Evaluation of Translation 

Before discussing precise error weighting I 
should say a few words about linguistics, dreary 
as the prospect may be, and follow with some 
comments on how committees apply linguistic 
principles in test evaluation. 

The first point to be made is that the grad- 
ing system to be described is based insofar as 
possible on a somewhat artificial distinction 
between syntax and bmavvUcA. I will therefore 
define these terms insofar as they apply to the 
present case. 

• Syntax is the system of relating linguistic 
elements to one another in phrase, clause, 
or sentence, a process which is concerned 
with selecting appropriate grammatical 
forms (hereafter called "affixes"), posi- 
tions for vocabulary items, or a combina- 
tion of both, depending upon the particu- 
lar language. Syntactic analysis leads 
from particular texts to general statements 
about the linguist elements of a language. 
The traditional parsed sentence is an in- 
ductive exercise proceeding from, for ex- 
ample, the sentence "The quick brown fox 
jumped over the lazy dog" to a set of names 
or symbols : Definite article + adjective + ad- 
jective + noun + verb + past tense + preposi- 
tion + definite article + adjective + noun. 
We cannot, however, recapture this gem by 
simply reversing the procedure, because, 
except for the definite article, a possibly 
infinite number of combinations could ap- 
pear. Nonetheless, a command of the syn- 
tax of a foreign language, which will in- 
evitably differ from that of English at 
many points, is the first element of the 
language the student necessarily learns. 

* Semantics, on the other hand, is concerned 
with particular meanings of linguistic ele- 
ments in context. To the extent that pat- 
terns of feeling and thinking between lan- 
guages are similar, as to a large extent 
they will be in the Indo-European languages, 



transfer of ideas and impressions will be 
on the whole fairly simple. (Some kinds of 
texts are dependent on an intimate know- 
ledge of a vastly different culture sub- 
system, religious, political, and so on. 
These texts are never used for PQEs.) The 
student needs a vocabulary of a couple of 
thousand words, the main carriers of ideas, 
before he can work efficiently, regardless 
of how good his syntactic control may be. 

Obviously, many members of PQE Committees 
have not had much formal linguistic training in 
syntactic and semantic analysis, excellent 
though they may be in the language they are con- 
cerned with. Just as obviously, there is no 
time for us to give extensive training in these 
subjects to people who are busy with other things. 
However, a certain minimum of linguistic think- 
ing is needed to make our system work and we 
have tried to impart this informally to commit- 
tees before they select passages for translation 
and evaluate them. Once a committee has a gen- 
eral grasp of the system, the members should be 
better able to select passages containing inter- 
esting syntactic and semantic problems. After 
having produced translations of passages against 
which papers will be judged, the committee can 
then to some extent predict likely errors. When 
the papers are actually graded, each evaluator 
works independently to derive a score based on 
the weighting described below then compared his 
results with the others. 

Some differences in weighting are bound to 
occur and may be irreconcilable, but generally 
the findings show a narrow spread of 5 to not 
more than 10 points. The care required in such 
careful grading insures that at the very least 
major errors are not overlooked, a result not 
always achieved by subjective grading. 

Our weighting of linguistic elements is based* 
on the assumption that aspirants should be ex- 
pected to control their source- language syntax. 
A failure to understand relations within and 
among sentences, which are marked by position 
and/or affixes as noted above, can seriously 
distort the meaning of an entire paragraph; 
hence, we take either 8 or 4 points off per er- 
ror, depending on its severity. If, for exam- 
ple, a sentence should have been translated 
"Dog bites man" but comes out "Man bites dog, " 
we would subtract 8 points because all the re- 
lationships are wrong. If a sentence which 
ought to have been translated "We stopped near 
the train" is rendered "We stopped the train, " 
we would take off 4 points since the subject- 
verb relationship is correct even though the 
rest of the sentence is wrong. As we have set 
70 as a passing score, four 8-point errors will 
fail a PQE. 

Certainly the examples above are oversimpli- 
fications of translation problems which can be 
knotty. Our experience has shown, however, that 
in 9 out of 10 cases committees can achieve con- 
sensus on the kind and degree of syntactic error 
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not only for Russian examinees but also for as- 
pirants in Indo-European and Semitic languages 
generally. Where we have had problems, it has 
been in languages such as Chinese or Vietnamese 
where the intersection of "syntax" and "semantics" 
has not begun to be explored, let alone taught 
to unversity or DLI students. If aspirant trans- 
lations from these languages go far afield and 
reverse the meaning of the original, or, even 
worse, do not even fall in the correct subject- 
matter area, the point system above is still ap- 
plied: wrong is wrong. It would of course be 
possible to set as passing a score lower than 70 
for languages structurally removed from English, 
but a decision to do so would be made on grounds 
other than linguistics. 

Semantic errors are "charged at a lower rate" 
(2 points per error) , not because we regard vo- 
cabulary as unimportant — far from it. As I 
noted above, a knowledge of syntax alone won't 
do, for the idea structure mainly depends upon 
lexical units. However, our experience has shown' 
that students of foreign languages pick up vo- 
cabulary unevenly, probably because no systematic 
attempt to teach it has ever been devised and 
successfully applied. Nor, for that matter, are 
students properly instructed in the use of either 
monolingual or bilingual dictionaries. To the 
extent that vocabulary instruction is given at 
all, it is given on the job and takes the form 
of ad hoc corrections and suggestions on how to 
use one or another dictionary to better effect. 
This process may be tedious but, provided the 
subject matter translated is not abstruse, is 
easy in comparison with mastering a new syntax. 

For these reasons it seemed best for us to 
accommodate to the existing situation and apply 
the lighter penalty. Even so, we still encounter j 
situations in which it is almost impossible to 
decide whether an error is in fact best viewed 
as semantic or syntactic. If a source- language 
pronominal reference is translated by the wrong 
pronoun, is this a simple error at a given point 
(hereafter, "punctilinear error") or could it 
open the way for misunderstanding several sen- 
tences (syntactic error)? Contextual judgment 
is the only method for making such a decision. 
On the other hand, some apparently pure grammar 
markers, such as tense forms, can be mistrans- 
lated without in any way distorting the basic 
noun-verb relationships. Thus, "the dog bites 
the mzn" and "the dog bit the man" have the same 
noun-verb-noun arrangement: the difference lies 
in the time of the event identified at a given 
point in the text. A tense error is hence punc- 
tilinear and costs 2 points. 

As I have suggested, it is often difficult 
for PQE Committees to reach a consensus at 
every point of semantic and syntactic error in 
translation from the source language. The task 
of evaluating the translation from the point of 
view of English usage and violation of conven- 
tion is even harder, but extremely awkward Eng- 
lish does have an overall effect of blurring 



meaning or, at the very least, causing the 
reader unnecessary strain. Violation of accep- 
table English we call "poor cooccurrence." 

When the members of a committee are unani- 
mous in judging an English phrase to reflect 
poor cooccurrence, one point is assessed. Here 
are examples of the kind of poor usage we have 
met: "an oblivious scholar to practical matters" 
instead of "a scholar oblivious to. . ." (awk- 
ward splitting of adjective and accompanying 
preposition); "the car collided against the 
bus" instead of ". . , with the bus" (a seem- 
ingly unlikely mistake for a native speaker of 
English to make, but this type is of high fre- 
quency nonetheless) ; "Foreign Minister Rogers" 
instead of "Secretary of State Rogers" (a vio- 
lation of conventional usage) . 

Hundreds, if not thousands, of similar pec- 
cadilloes have been and are being committed un- 
der the present system and are the object of 
heated remonstrances from the failing aspirant. 
("Why do they have to nitpick everything?") It 
has been our personal experience that commit- 
tees on the whole err on the side of generosity 
in cases of one-point errors, and overlook al- 
together misspellings and wrong punctuation 
where these do not cause major problems in un- 
derstanding the English text. Even those eval- 
uators who take violent exception to misspelling 
will not fail a paper for this alone. 
* 

To sum up, it has been our intention to es- 
tablish a system of grading which will be as 
objective as possible. Using a base of 100 and 
with 70 as passing, we have specified 3 kinds 
of errors. The first 2, syntactic and semantic, 
concern command of the source language; the 
third bears upon English usage. Syntactic 
errors are of two orders of severity reflected 
in the 8- and 4-point penalties, while semantic 
mistakes are charged 2 points and English mis- 
use, or poor cooccurrence, costs one. 

There are, to be sure, many doubting Thomas- 
es who question the efficiency of the system, 
either because they feel it is too complex to 
master or that the time spent in mastering it 
might better be devoted to something else and 
that the intuitive or "seat-of-the-pants" method 
will do as well. While conceding the time re- 
quired to be considerable, the final result, we 
feel, more than justifies the effort and, in the 
absence of a better system, ought to prevail. 



Editor's note : The system described 



in the reprinted article has be en in use 
at the Agency since July 1972. [~ 



is currently wr^tvng a paper vn wh%ch he 
will summarize the experience that the 
PQE Committees have had in using the sys- 
tem and will suggest theoretical and 
practical improvements. 
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How does an NSA language professional stack 
up against language professionals elsewhere? 
That question cannot really be answered inas- 
much as apples and pineapples are not directly 
comparable. An insight at the very lowest level- 
of professionalization can be gained, however, by 
comparing the criteria for NSA's open-source 
translation {Part IA of the Professional Quali- 
fication Examination) and those for accreditation 
by the American Translators Association (ATA). 

NSA* 8 test 

The NSA language test is of such length in the 
foreign language as to generate a total of 450 to 
600 words of English translation. Time is not a 
factor, therefore examinees can take as long as 
4 hours for this section of the PQE. Texts do 
not pose any problems of content substance; that 
is, they do not require an overly specialized 
knowledge of subject matter. They are usually 
articles by professional journalists on a topic 
of interest to the general public --at the dif- 
ficulty level of Reader's Digest or a weekly 
news periodical. Any lexical item in the text 
which is not in the standard bilingual diction- 
ary is glossed in a footnote. 

NSA graders use a points-off system in grad- 
ing the test. They give more relative weight 
to errors involving the logical relationships 
among major sentence parts and to identifying 
specific items in the source text (that is, 
markers for number, case, gender, tense, aspect, 
mood, voice, etc.). Each sentence is graded by 
itself; no more than 8 points is taken off for 
any one sentence. If the logical subject-verb- 
object/complement relationship in the translation 
is not substantially the same as that of the 
source- language sentence, no fewer than 4 and no 
more than 8 points are taken off. Experience in 
various test committees has shown that examinees 
who get the maximum of 8 points taken off on 
each of 3 sentences seldom pass the test. 



For each translation error involving a 
single word (content words such as nouns, 
adjectives, verb stems; function words such 
as prepositions and conjunctions, and gram- 
matical affixes) 2 points are deducted -- 
up to a maximum of 8 points in any one 
sentence. For errors involving poor selec- 
tion of dictionary meanings of a lexical 
item or poor word collocations (words that 
just don't fit together right in English), 
only one point is deducted. 

From the foregoing explanation, it can be 
seen that a sentence in the English translation 
might contain one major error involving logical 
relationships (say, 8 points) , 3 single-word 
errors (6 points), and perhaps 2 collocation or 
word-choice errors (2 points) , but the NSA 
gTader would not deduct 16 points for that sen- 
tence -- instead he would deduct only the maxi- 
mum of 8 points per sentence. 

If the translation of Part IA of the Agency's 
PQE has fewer than 30 points deducted, it is 
considered to be a "pass." But a translation 
that just passes is far from good. It is not as 
simple as saying that a translation with 30 
points deducted is "30% inaccurate, " Instead, 
such a translation may be "all wrong," because 
it will almost certainly contain several errors 
of fact, any one of which could be critical in 
a real -life translation situation. So, if 
such a poor examinee were to be certified as a 
linguist and were to produce, on an everyday 
basis, operational translations containing 
such critical errors of fact, one can only 
imagine the tremendous amount of work and re- 
sponsibility that would be placed on the lan- 
guage checker who would have to correct the 
translations before issuing them. Or on6 could 
imagine the inherent dangers if such transla- 
tions, in a stress situation, were issued to 
the customer without being corrected. 
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If a particular grader deducts between 30 
and 40 points, the translation must be debated 
in committee; if the majority of the committee 
members feel that enough of the minor errors 
would have been caught and corrected in the 
checking, proofreading, and analysis phases, 
a translation with as many as 40 points might 
actually be allowed to pass. PQE committee 
members sometimes lean over backwards in at- 
tempting to pass certain marginal papers. They 
claim that if only a few examinees pass, man- 
agement tends to say that the test is too hard, 
never that the applicants are just not certifi- 
able as professional linguists. 

One thing must be borne in mind, however. 
A person who passes the PQE, even with distinc- 
tion, is only certified as a linguist; it still 
might take him many years of on-the-job experi- 
ence to become a senior linguist (something 
that a person becomes without having a piece 
of paper to prove it). Dr. Jaffe used to com- 
pare this situation with that of a medical stu- 
dent being awarded a diploma from medical 
school and the right to apply for a license to 
practice medicine; he would still need years of 
experience to become an expert in his field. If, 
then, a person who passes a PQE without debate 
has demonstrated the potential to become a 
senior linguist, the person who is allowed, 
after debate, to squeak through a PQE has de- 
monstrated, if anything, the ability to function 
at only the very lowest level of linguistic 
competency. 

ATA test 

The American Translators Association is a 
different animal. Its members include both 
literary translators and technical translators. 
Literary translation is a creative art, and is 
not tested. But since all translators make 
most of their money from translating technical , 
scientific, and legal material, technical trans- 
lation is used for ATA accreditation. The 
length of the ATA test is" approximately 750 
words. Time is a factor; precisely 3 hours is 
allotted for the test. As with the NSA text, 
the texts do not pose any problems of content 
substance: lexical items not in the standard 
bilingual dictionary are glossed in footnotes. 
Selections tend to reflect the subject matter 
of professional translators (international 
meetings, symposia, national projects, medical 
research, technology, science, etc.), but in a 
popularized version at the newspaper or Reader's 
Digest difficulty level. 

Backgrounds and specializations of the ATA 
members vary greatly. The examinees are ac- 
cordingly offered five selections in five some- 
what different subject fields, from which they 
choose three to translate. The topics in the 
test that I recently took were the kpollo-Soyuz 
flight, the Helsinki Conference, blood immunolo- 
gy, a legal case involving patents, and the 
prospective Canadian Health Service System. 



These selections do not reflect the diffi- 
culty level of the foreign- language texts that 
usually confront professional translators. But 
the grading criteria do reflect the requirement 
of the customers who let contracts for trans- 
lators. The ATA examinees are told to be pre- 
cise and fairly literal, and to be extremely 
carefuly in paraphrasing so as not to lose one 
iota of the sense of the clause or sentence. 
They are warned that omissions of words (except 
for such low- information words as schon, doch y 
und zwar, en effet s znackit) can be extremely 
costly and can even result in failing a selec- 
tion. A candidate must pass two out of the 
three passages selected. Only one serious er- 
ror in a passage is permitted; two serious er- 
rors constitute a fail for that selection (of 
course, a large number of nonserious errors 
could also constitute a fail). 

Following is a passage from a statistical 
text, Calaul des probabilitts 9 that was used in 
the French test administered at the ATA work- 
shop in California in the summer of 1975. The 
passage is followed, in turn, by three of the 
translations done by examinees (for the con- 
venience of the grading committee, the transla- 
tions are typed prior to grading) . The under- 
lined words and the notations in the left-hand 
margin were made by one of the ATA graders (a 
capital E is the symbol for a major error; a 
lower-case e is the symbol for a minor error; 
'mistr. stands for mistranslation.) 

Comparison of ATA and NSA grading systems 

To compare the ATA and NSA grading systems, 
the typed translations, including the ATA grad- 
er's underlines and notations, were shown to an 
NSA linguist with experience in grading French 
PQEs. He made his own underlines (======), 

most of which coincided with errors noted by 
the ATA grader, and indicated in the right-hand 
margin the number of points that would be de- 
ducted in accordance with the Agency's PQE 
criteria. For example, in translation #1 
the ATA grader considered the error "for both" 
(mistranslation of pour les dieux 3 "for the 
gods," misread as pour les deux) to be a major 
error, whereas the NSA grader deducted 2 points 
for it. 

Since the Agency's PQEs are usually twice as 
long as the ATA French selection, let us assume 
that the translators would have had an equal 
error rate for an additional 300 words. In that 
event, the results would have been as follows: 



NSA grade 



Titans - 
lator 

1 
2 
3 



ATA 
grade 

Fail 
Fail 
Pass 



Raw 
score 

84 
72 
100 



Extrapolated 
score 

68 
44 
100 



Pass/ 
fail 

Marginal 

Fail 

Pass 
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French, passage A -- from a statistical 
text, Calcul des probabilitis 



Translator #1 



I. LE HASARD 

"Comment oser parler des lois du hasard? 
Le hasard n'est-il pas l'antithese de toute 
loi?" Ainsi s'exprime Bertrand, au debut 
de son Calcul des probabilitis • La probabi^ 
lite est opposee a la certitude; c'est done 
ce qu'on ignore et, par consequent semble- 
t-il, ce qu'on ne saurait calculer. II y a 
la" une contradiction au moins apparente et 
sur laquelle on a deja beaucoup ecrit. 

Et d'abord qu'est-ce que le hasard? Les 
anciens distinguaient les phenomenes qui 
semblaient obeir a des lois harmonieuses , 
etablies une fois pour toutes, et ceux qu'ils 
attribuaient au hasard; e'etaient ceux qu'on 
ne pouvait prevoir parce qu'ils e'taient 
rebel les a toute loi. Dans chaque domaine 
les lois precises ne decidaient pas de tout, 
elles tra^aient seulement les limites entre 
lesquelles il etait permis au hasard de se 
mouvoir. Dans cette conception, le mot 
hasard avait un sens precis, objectif: ce 
qui etait hasard pour l'un, etait aussi 
hasard pour 1* autre et meme pour les dieux. 

Mais cette conception n'est plus la 
notre; nous sommes devenus des deterministes 
absolus, et ceux memes qui veulent reserver 
les droits du libre arbitre humain laissent 
du moins le determinisme regner sans partage 
dans le monde inorganique. Tout phenomene, 
si minime qu'il soit, a une cause, et un 
esprit infiniment puissant, infiniment bien 
informe des lois de la nature, aurait pu le 
prevoir des le commencement des siecles. Si 
un pareil esprit existait, on ne pourrait 
jouer avec lui a aucun jeu de hasard, on 
perdrait tou jours. 

Pour lui, en effet, le mot de hasard 
n 1 aurait pas de sens, ou plutot il n'y 
aurait pas de hasard. C'est a cause de 
notre faiblesse et de notre ignorance qu'il 
y en aurait un pour nous. Et, meme sans 
sortir de notre faible humanite, ce qui est 
hasard pour 1' ignorant, n'est plus hasard 
pour le savant. Le hasard n'est que le 
mesure de notre ignorance. Les phenomenes 
fortuits sont, par definition, ceux dont 
nous ignorons les lois. 



(3 



I. CHANCE 

"S—^f^— JP^g*L of the laws Of 
chance? "is not cKance the antithesis of 
every law?" Thus Bertrand expresses him- 
self, at the beginning of his "Calculation 
of Probabilities. " Probability is op- 
£ , .^posed to certainty; thus it is what is un- £v 
C'^'^fy; ^nown , /S what we would not know how to 

^ m calculate. There is at least an apparent 
contradiction here , one on which a great 
deal has already been written. 

- First off, what is chance? The ancients 

distinguished between phenomena ()£ 

which seemed to obey harmonious laws, es- 
tablished once for all, and those which 
they attributed to chance; they were the 
onesr that couldn 1 1 be anticipated because 
i:they were unamenable to all law. In each 
domain, no precise laws were determined ; fin 

they traced only the boundaries among 
which chance was allowed to operate. In 
this concept, the word "chance" had a pre- 
cise, objective meaning: what was chance 
for one was also chance for the other , and 
even for both. A 



r* 1 



However, this is not our way of looking 
at it_£.we have become absolute determin-% 
ists , and even the same ones who want to ^ Q 
save a place for free Will in mankindO vrj^ 
neverthelesa^)let determinism hold un- 
trammeled sway in the inorganic world. 
Every phenomenon, no matter how minute it 
may be, has a cause, and an infinitely pow- 
erful mind, infinitely well informed of the 
laws of nature, would have been able to fore- 
tell it since the beginning of time. If such 
a mind existed, one would not be able to play 



,j*»-A ■ any games of chance with 



— one would 



always lose. 

For that mind, in fact, the word "chance" 
would have no meaning, or rather there would 
be no chance . It is because of our weakness 
and our ignorance that there would be chance 
H"' with us . Also , even without"gb"fng"outiide^ 
o? = o = ur = f eeble humanity, what is chance to V. 
the ignorant is no longer chance to the wise 
man. Chance is only the measure of our igHM 
norance. T^fe chance phenomena , by defi-Q 
nition, are thdse~whose laws we do not know. 
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Translator #2 



Translator #3 



I. CHANCE 

"How does one dare to speak of the 
laws of chance? Isn't chance the antithe- 
sis of all law?" Bertrand thus expresses 
himself, at the beginning of his "Calcula- 
tion of Probabilities . " Probability is 
opposed to certainty; it is thus what one 
<ls does not know and, consequently, it seems, 
tffi**' what one does not know how to calculate . (5) 
There is~ there~ at~lealt~an" apparent con-^ 
tradictio nj a^d one on which much has 
already been written. 

J^[tff\ A^t , first, what is chance? The an- CD 



singled = out phenomena that seeme^T) 
to obey harmonious , firmly established 
laws and those they attributed to chance; 
^ TV there were those that one couldn't forsee^) 
becaule"~they were £gbellious to all 
law. In each domain, precise laws did not 
decide everything, they only traced the 
limits between which chance was permitted 
to operate. In this conception, the word 
"chance" had a precise, objective meaning: 
that which was chance for one, was also 
chance for the other and even for the gods . 

But this idea is no longer ours; we have 
become absolute determinists , and even 
those who wish to reserve the laws of human 
free-will ^leave^tr leas^ tfc£ determin- 
ism to rule without division in the in- f£} 



tft^organic world. All phenomenum7,^ flSL^ no| 
as small as it is, has a cause, and an 



£ infinitely powerful spirit , infinitely 0/ 
*f " well formed to the laws~ol~hature , would (§) 
^. ii, have Been~aBfe to f orsee it from the be- 
&yy. ginning of the centuries. If a like 

splrlt ~exlste571^b~game"bf chance" could 
1^ be played with it, one would always lose. 

For it, indeed, the word gf chance Q) 
Jj^tf would not have any sense , or soon there 

£ V* v would not be any chance. It is because 

of our weakness and our ignorance that 
^pr. there would be one for us . And , even 

without" leaving our weak humanity, Q 

that which is chance for the ignorant, is 
no longer chance for the well-informed. 
Chance is only the measure of our igno- 
^ ranee. Fortuitous phenomena are, by 
^\<>Tf- definition, those of which we do not 
know the laws. ===== ^--^ 



I. CHANCE 

"How can we dare to speak of the 
laws of chance? Is not chance the anti- 
thesis of all law?" Thus speaks Bertrand, 
at the beginning of his "Calculation of 
Probabilities. " Probability is opposed 
to certitude; thus it is what we do not 
know and so, apparently, what we cannot 
calculate. There is here an at least 
apparent contradiction about which much 
has already been written. 

First of all, what is chance? The 
ancients distinguished between phenomena 
which seemed to obey harmonious, defi- 
nitely established laws and those which 
they attributed to chance. The latter 
were phenomena that could not be foreseen 
because they were contrary to all law. 
In each field the precise laws were not 
all-determining; they merely defined the 
limits within which chance was permitted 
to operate. In this conception, the 
word "chance" had a precise, objective 
meaning: what was chance for one man was 
chance for another, and even for the gods . 

But this conception is no longer ours; 
we have become absolute determinists, and 
even those who wish to reserve the rights 
of human free will at least permit deter- 
minism to reign unchallenged in the inor 
ganic world. Every phenomenon, however 
small, has a cause, and an infinitely 
powerful mind, infinitely well informed 
of the laws of nature, could have fore- 
seen it from the beginning of time. If 
such a mind existed, we could -not play 
any game of chance with it; we should 
always lose. 

For this mind, in fact, the word 
"chance" would have no meaning, or rather 
there would be no chance. It is because 
of our weakness and our ignorance that 
one may exist for us. Even without de- 
parting from our weak humanity, what is 
chance for the ignorant man is no longer 
chance for the scientist. Chance is 
merely the measure of our ignorance. 
Fortuitous phenomena are, by definition, 
those whose laws we do not know. 
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One final remark about NSA PQEs and ATA's 
accreditation examinations: they are both de- 
signed to set the lowest threshold level of 
professionalization, no morel At NSA the em- 
phasis is placed on accuracy of content. In 
the ATA, accent is placed on precise, literal 
accuracy and on speed. A person who can com- 
plete the exam in exactly 3 hours will be able 
to make a bare living just above the poverty 
level in translating material of that difficul- 
ty, using conventional paper -and -pencil tech- 
niques and typing up the finished copy himself. 
Translators therefore specialize in specific 
technical fields where they can get at least 
double or triple the basic translation pay 
rate. Then too, professional translators often 
dictate their translations into recording 
equipment and pay a typist to transcribe the 
translation into finished copy; despite this 
additional expense, this method is beneficial 
to the translator since it enables him to at 
least double his translation output. Conse- 
quently, a free-lance professional translator, 
after gaining experience and one or more 
specialties, should be able to earn at least 
$36,000 a year. Even a part-time free-lance 
(moonlighter) translator can earn as much as 
$5000 a year. These figures indicate that ATA 
accreditation is worth quite a bit in dollars 
and cents. They also indicate that, according 
to ATA standards, the majority of certified NSA 
linguists are overpaid, while the few truly 
professional certified linguists at NSA are 
underpaid. 
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Chief, W 



24 December 1975 



TO: 



B 



SUBJECT : Cryptoloqr Article , Oct 75 

In one of the gaps which occurred dur- 
ing the holiday season I got to read your 
"...Proud and Bitter Memories article. Your 
message is clear, as are your feelings about 
the people you worked with (and for")-, I 
think you've done a service for those of us 
here who didn't have much to do with "the . 
problem" by writing that article. It wasn't 
enjoyable reading, but I'm recommending it 
to all in W. 



cc: Ch f B 
Ch, PI 




Editor, Cryptolog, P16 



1+ 



Communications Security 
Establishment 



I very much enjoyed the October issue of 
Cryptolog, especially ! I "Proud 

and Bitter Memories" and the extract from 
his "Uncertain Origins" which you put 
across as a splendid Double-Crostic. Not- 
ing that you are obviously very enthusi- 
astic about this kind of puzzle, I am en- 
closing a copy of a related type which I 
constructed for fun, but on completion 
realized was not economically viable, since 
it is much more difficult and time-consum- 
ing to create than to solve. . . I don't 
think I shall ever construct another one, 
so this is probably an authentic O'Neill 
hapax legomenon, . . 

Kevin O'Neill 



Readers who would like a copy of 
Mr. O'Neill's puzzle may get one 
by calling the Editor on 5642s. 

(UNCLASSIFIED) 



P.L. 86-36 
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